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ABSTRACT 


;  -  Huang  and  Hsu  [1]  describe  an  image  sequence  enhancement 

algorithm  based  on  computing  motion  vectors  between  successive 
frames  and  using  these  vectors  to  determine  the  correspondence 
between  pixels  for  frame  averaging.  In  this  note,  we  demon- 

(strate  that  it  may  be  sufficient  to  use  only  the  components  of 

the  motion  vectors  in  the  gradient  direction  (called  the  nor¬ 
mal  components)  to  perform  the  enhancement. 
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/  In  a  recent  paper,  Huang  and  Hsu  [ 1 3  describe  an  enhance¬ 
ment  algorithm  for  time-varying  images.  This  algorithm  achieves 
the  type  of  enhancement  that  one  would  obtain  from  averaging 
multiple  images  of  a  static  scene  by  using  motion  vectors  com¬ 
puted  from  the  sequence  to  establish  the  correspondence  between 
pixels  in  successive  frames. 

The  purpose  of  this  note  is  to  demonstrate  how  and  why  it 
is  often  sufficient  to  consider  only  the  normal  components  of 


the  motion  vectors,  i.e. ,  the  component  in  the  gradient  direc¬ 
tion,  to  compute  such  enhancements.  There  is  a  computational 
advantage  to  using  only  the  normal  component,  since  it  is 
directly  measurable  from  the  image  sequence  by  combining  the 
space  and  time  derivatives  of  image  intensity,  while  computing 
the  actual  motion  vectors  ordinarily  requires  either  some  spa¬ 
tial  integration  of  the  normal  components  or  the  application  of 
a  local  correlation  procedure.  Furthermore,  in  order  to  compute 
the  motion  vectors  one  must  make  some  assumption  about  the  local 
image  motion  (for  example,  that  the  image  motion  is  a  two-dimen-' 
sional  translation);  when  this  assumption  is  incorrect,  the  com¬ 
puted  motion  vectors  will  be  in  error.  ~ — - - 

Using  the  motion  vectors  to  perform  image  sequence  enhancement 
implies  that  it  is  important  to  establish  the  exact  correspondence 
between  points  from  frame  to  frame.  In  fact,  if  we  assume  that 


the  objects  in  the  image  sequence  have  (piecewise)  slowly  varying 


intensity,  then  it  is  sufficient  to  simply  determine  a  corre¬ 
spondence  between  nearby  points  that  belong  to  the  same  image 
region,  since  their  intensities  should  be  quite  similar.  The 
normal  component  can  be  used  to  establish  such  a  correspond¬ 
ence.  Indeed,  let  P  be  a  pixel,  let  P*  be  the  result  of  dis 
placing  P  by  the  true  motion  vector,  and  let  P'  be  the  result 
of  displacing  it  by  the  normal  component  only.  Thus  P*  is 
the  true  position  of  P  under  the  motion;  but  P'  differs  from  P* 
only  by  the  (unknown)  tangential  component  of  the  motion,  and 
so  is  likely  to  belong  to  the  same  region  as  P*.  The  concept 
here  is  analogous  to  that  used  in  early  work  on  smoothing  of 
single  images  [2] ,  where  a  pixel  at  which  the  gradient  magni¬ 
tude  is  high  is  smoothed  by  averaging  with  its  neighbors  in 
the  tangential  direction  only. 


t 


2.  Enhancement  algorithm 


Let  and  f 2  be  two  frames  from  a  motion  sequence,  and  let 
m  be  the  field  of  normal  components  of  the  motion  vectors  be¬ 
tween  f^  and  f2*  The  picture  m  is  a  vector-valued  picture, 
where  m(x,y)  =  (u,v);  u  is  the  projection  of  the  normal  compo¬ 
nent  on  the  x-axis  and  v  is  the  projection  of  the  normal  com¬ 
ponent  on  the  y-axis.  We  will  let  u(x,y)  and  v(x,y)  denote 
the  u  and  v  components  of  m  at  (x,y). 

We  compute  an  enhanced  version  of  f^,  which  we  will  denote 
as  g2/  by  the  following  two  stage  process: 

1)  Create  a  "shifted"  version  of  f^,  denoted  f|,  as  follows: 

a)  Initially,  for  all  (x,y)  set  f|(x,y)=U  {for  "unde¬ 
fined"  ) 

b)  For  all  (x,y)  set  f | (x+u (x,y) ,  y+v  (x,y)  (x,y ) 

2)  Create  the  enhanced  version  of  f2  as  follows: 

For  all  (x,y) 

If  f^(x,y)=U  Then  g2 (x,y )=f 2 (x,y) 

Else  g2 (x,y)= [ f£ (x,y)  +  f2(x,y)]/2 
In  step  (1)  the  pixels  having  no  predecessor  in  the  image  under 
the  motion  are  assigned  the  value  "U"  in  the  shifted  version  of 
f^.  At  these  positions  we  cannot  compute  a  correspondence  be¬ 
tween  pixels  in  f1  and  f2,  so  that  we  use  the  intensities  in  f2 
to  form  the  enhanced  image.  Also,  it  is  quite  possible  that  sev¬ 
eral  points  in  f^  map  onto  the  same  point  in  f^  due  to  errors  in 
the  normal  components.  Note  that  this  could  also  happen  if  we 
used  the  motion  vectors  rather  than  only  the  normal  components. 


The  above  algorithm  can  be  simply  extended  to  compute 
enhancements  based  on  sequences  of  more  than  two  frames.  Let 
f . ,  f0,  . ..,  f  be  the  sequence  of  frames  for  which  we  wish 
to  produce  an  enhanced  image.  We  proceed  as  follows:  Let 
m^  be  the  normal  component  or  motion  vector  field  for  the  ith 
frame.  Apply  the  above  algorithm  to  frames  f^  and  f2  to  pro¬ 
duce  an  enhanceu  image  of  those  two  frames,-  which  we  will  de¬ 
note  by  912"  Next«  apply  the  enhancement  algorithm  to  g12  and 
f3  (using  the  motion  vector  field  n»2  computed  from  the  initial 
sequence)  to  produce  an  enhanced  version  of  f^-f^  denoted  g^3. 
Continuing  in  this  manner,  we  can  produce  an  enhanced  version  of 

frames  f.  to  f  . 
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A  problem  with  this  approach  is  that  as  we  compose  the  motion 
vector  fields  in  moving  through  the  sequence  the  errors  in  the 
individual  motion  vector  fields  accumulate,  so  that  gin  is  not 
a  uniformly  good  enhancement  of  to  fn-  Instead  we  adopted 
the  following  "triangular"  procedure  which  takes  advantage  of 
the  intermediate  computations  that  would  have  to  be  done  to 
compute  enhancements  cf  subsequent  sequences  (e.g.,  f2  to  fn+^) 
for  computing  the  enhancement  of  to  fn«  Instead  of  simply 
regarding  g^n  as  the  enhanceu  version  of  f^  to  fn»  we  compute 
the  median  of  i=l,n-l  and  fn  as  the  enhancement.  The 

images  g ^n,  i=2,n-l,  are  intermediate  results  in  the  computation 
of  g^^  i+n_i  and  30  “UISt  be  computed  in  any  event.  We  found  ex¬ 
perimentally  that  median  filtering  results  in  better  enhancements 
than  simple  averaging. 


3.  Experimental  results 


Figures  1  and  2  show  two  image  sequences,  each  containing 
8  frames.  Sequence  1  contains  images  of  a  printed  page  which 
was  moved  in  front  of  the  camera  parallel  to  the  image  plane 
(thus  the  motion  was  a  two-dimensional  motion) .  White  noise 
was  then  added  to  each  frame  in  the  sequence.  Sequence  2  con¬ 
tains  images  of  a  toy  with  significant  detail.  The  camera  was 
rotated  so  that  the  image  of  the  toy  moved  from  left  to  right 
in  the  sequence,  and  in  this  case  the  motion  is  a  three-dimen¬ 
sional  motion. 

The  results  comparing  enhancements  obtained  using  the  normal 
components  of  the  motion  vectors  versus  the  actual  motion  vec¬ 
tors  are  displayed  in  Figures  3  and  4.  The  motion  vectors  and 
normal  components  were  computed  using  a  differential  technique. 
Spatial  and  temporal  intensity  derivatives  were  approximated  based 
on  cubic  spatial  and  temporal  approximations  to  sequences  of  five 
frames,  with  the  motion  vectors  associated  with  the  third  frame 
in  each  sequence  of  five.  The  motion  vectors  were  derived  from 
the  normal  components  under  the  assumption  that  the  motion  was 
locally  a  simple  image  plane  translation  (see,  for  example  [3,4, 
5)).  For  comparison,  enhancements  based  on  both  averaging  and 
median  filtering  (see  the  discussion  at  the  end  of  the  previous 
section)  are  displayed,  as  well  as  enhancements  without  motion 
compensation.  The  results  using  the  motion  vectors  and  the  normal 
components  are  quite  comparable,  and  indicate  that  it  may  not  al¬ 
ways  be  necessary  to  incur  the  additional  expense  of  determining 
the  true  motion  vectors  to  perform  enhancement. 


4.  Conclusions 

In  this  paper  we  have  demonstrated  that  it  may  not  always 
be  necessary  to  compute  motion  vectors  to  enhance  image  se¬ 
quences,  and  that  instead  it  may  be  sufficient  to  consider  only 
the  normal  component  of  motion.  However,  we  should  note  that 
this  observation  may  depend  on  the  specific  enhancement  algo¬ 
rithm  chosen;  also,  since  the  evaluation  of  image  enhancements 
is  based  on  visual,  subjective  criteria  we  should  point  out 
that  Huang  (1]  remarked  that  the  enhancements  are  stronger  when 
the  enhanced  images  are  themselves  viewed  as  an  image  sequence. 
We  were  unable  to  perform  such  a  "dynamic"  comparison  with  nor¬ 
mal  component  enhancements,  but  the  results  of  such  a  subjective 
comparison  would  be  important  in  deciding  whether  or  not  to  use 
just  the  normal  components  for  enhancement  . 
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